To reduce the high computational complexity of intra coding in High Efficiency Video Coding (HEVC), a fast intra depth decision algorithm for Coding Unit (CU) based on spatial correlation property of images was proposed. First, the depth of the current Coded Tree Unit (CTU) was estimated by linearly weighting the adjacent CTU depths. Then appropriate double thresholds were set to terminate the CTU splitting process or skip some depths of CTU, thereby reducing unnecessary depth calculation. Experimental results show that compared with HM12.0, the proposed intra-depth decision optimization algorithm can significantly decrease the coding time of simple video sequence with only negligible drop in quality, when the Y-PSNR dropped by an average of 0.02 dB, the encoding time is reduced by an average of 34.6%. Besides, the proposed algorithm is easy to be fused with other methods to further reduce the computational complexity for HEVC intra coding, and ultimately achieves the purpose of real-time transmission of high-definition video.